On Falsification of Large-Scale Cyber-Physical Systems

In the development of modern Cyber-Physical Systems, Model-Based Testingof the closed-loop system is an approach for finding potential faults andincreasing quality of developed products. Testing is done on many differentabstraction levels, and for large-scale industrial systems, there are severalchallenges. Executing tests on the systems can be time-consuming and largenumbers of complex specifications need to be thoroughly tested, while manyof the popular academic benchmarks do not necessarily reflect on this complexity.This thesis proposes new methods for analyzing and generating test casesas a means for being more certain that proper testing has been performed onthe system under test. For analysis, the proposed approach can automaticallyfind out how much of the physical parts of the system that the test suite hasexecuted.For test case generation, an approach to find errors is optimization-basedfalsification. This thesis attempts to close the gap between academia and industryby applying falsification techniques to real-world models from VolvoCar Corporation and adapting the falsification procedure where it has shortcomingsfor certain classes of systems. Specifically, the main contributionsof this thesis are (i) a method for automatically transforming a signal-basedspecification into a formal specification allowing an optimization-based falsificationapproach, (ii) a new collection of specifications inspired by large-scalespecifications from industry, (iii) an algorithm to perform optimization-basedfalsification for such a large set of specifications, and (iv) a new type of coveragecriterion for Cyber-Physical Systems that can help to assess when testingcan be concluded.The proposed methods have been evaluated for both academic benchmarkexamples and real-world industrial models. One of the main conclusions isthat the proposed additions and changes to the analysis and generation oftests can be useful, given that one has enough information about the systemunder test. The methods presented in this thesis have been applied to realworldmodels in a way that allows for higher-quality products by finding morefaults in early phases of development

Falsification of Signal-Based Specifications for Cyber-Physical Systems

In the development of software for modern Cyber-Physical Systems, testing is an integral part that is rightfully given a lot of attention. Testing is done on many different abstraction levels, and especially for large-scale industrial systems, it can be difficult to know when the testing should conclude and the software can be considered correct enough for making its way into production. This thesis proposes new methods for analyzing and generating test cases as a means of being more certain that proper testing has been performed for the system under test. For analysis, the proposed approach includes automatically finding how much a given test suite has executed the physical properties of the simulated system. For test case generation, an up-and-coming approach to find errors in Cyber-Physical Systems is simulation-based falsification. While falsification is suitable also for some large-scale industrial systems, sometimes there is a gap between what has been researched and what problems need to be solved to make the approach tractable in the industry. This thesis attempts to close this gap by applying falsification techniques to real-world models from Volvo Car Corporation, and adapting the falsification procedure where it has shortcomings for certain classes of systems. Specifically, the thesis includes a method for automatically transforming a signal-based specification into a formal specification in temporal logic, as well as a modification to the underlying optimization problem that makes falsification more viable in an industrial setting. The proposed methods have been evaluated for both academic benchmark examples and real-world industrial models. One of the main conclusions is that the proposed additions and changes to analysis and generation of tests can be useful, given that one has enough information about the system under test. It is difficult to provide a general solution that will always work best -- instead, the challenge lies in identifying which properties of the given system should be taken into account when trying to find potential errors in the system

Evaluating Optimization Solvers and Robust Semantics for Simulation-Based Falsification

Temporal-logic based falsification of Cyber-Physical Systems is a testing technique used to verify certain behaviours in simulation models, however the problem statement typically requires some model-specific tuning of parameters to achieve optimal results. In this experience report, we investigate how different optimization solvers and objective functions affect the falsification outcome for a benchmark set of models and specifications. With data from the four different solvers and three different objective functions for the falsification problem, we see that choice of solver and objective function depends both on the model and the specification that are to be falsified. We also note that using a robust semantics of Signal Temporal Logic typically increases falsification performance compared to using Boolean semantics

Multi-Requirement Testing Using Focused Falsification

Testing of Cyber-Physical Systems (CPS) deals with the problem of finding input traces to the systems such that given requirements do not hold. Requirements can be formalized in many different ways; in this work requirements are modeled using Signal Temporal Logic (STL) for which a quantitative measure, or \emph{robustness value}, can be computed given a requirement together with input and output traces. This value is a measure of how far away the requirement is from not holding and is used to guide falsification procedures for deciding on new input traces to simulate one after the other. When the system under test has multiple requirements, standard approaches are to falsify them one-by-one, or as a conjunction of all requirements, but these approaches do not scale well for industrial-sized problems. In this work we consider testing of systems with multiple requirements by proposing focused multi-requirement falsification. This is a multi-stage approach where the solver tries to sequentially falsify the requirements one-by-one, but for every simulation also evaluate the robustness value for all requirements. After one requirement has been focused long enough, the next requirement to focus is selected by considering the robustness values and trajectory history calculated thus far. Each falsification attempt makes use of a prior sensitivity analysis, which for each requirement estimates the parameters that are unlikely to affect the robustness value, in order to reduce the number of parameters that are used by the optimization solver. The proposed approach is evaluated on a public benchmark example containing a large number of requirements, and includes a comparison of the proposed algorithm against a new suggested baseline method

Industrial Temporal Logic Specifications for Falsification of Cyber-Physical Systems

In this benchmark proposal, we present a set of large specifications stated in Signal Temporal Logic (STL) intended for use in falsification of Cyber-Physical Systems. The main purpose of the benchmark is for tools that monitor STL specifications to be able to test their performance on complex specifications that have structure similar to industrial specifications. The benchmark itself is a Git repository which will therefore be updated over time, and new specifications can be added. At the time of submission, the repository contains a total of seven Simulink requirement models, resulting in 17 generated STL specifications

Multiple Objective Functions for Falsification of Cyber-Physical Systems

Cyber-physical systems are typically safety-critical, thus it is crucial to guarantee that they conform to given specifications, that are the properties that the system must fulfill. Optimization-based falsification is a model-based testing method to find counterexamples of the specifications. The main idea is to measure how far away a specification is from being broken, and to use an optimization procedure to guide the testing towards falsification. The efficiency of the falsification is affected by the objective function used to evaluate the test results; different objective functions are differently efficient for different types of problems. However, the efficiency of various objective functions is not easily determined beforehand. This paper evaluates the efficiency of using multiple objective functions in the falsification process. The hypothesis is that this will, in general, be more efficient, meaning that it falsifies a system in fewer iterations, than just applying a single objective function to a specific problem. Two objective functions are evaluated, Max, Additive, on a set of benchmark problems. The evaluation shows that using multiple objective functions can reduce the number of iterations necessary to falsify a property

Applying valued booleans in testing of cyber-physical systems

In software testing, as in cyber-physical systems testing, test suites are traditionally developed by hand. In this work we consider one framework for putting the computer in charge of the testing instead: constrained random test case generation as supported by the tool QuickCheck. This is implemented by the use of Valued Booleans (VBools). VBools naturally allow for an extension of QuickCheck into cyber-physical systems, which is useful particularly since QuickCheck can perform shrinking of test cases. Shrinking is a technique to make test cases simpler while preserving failure

Using Valued Booleans to Find Simpler Counterexamples in Random Testing of Cyber-Physical Systems

We propose a new logic of valued Booleans for writing properties which are not just true or false but compute how severely they are falsified. The logic is reminiscent of STL or MTL but gives the tester control over what severity means in the particular problem domain. We use this logic to simplify failing test inputs in the context of random testing of cyber-physical systems and show that it improves the quality of counterexamples found. The logic of valued Booleans might also be used as an alternative to the standard robust semantics of STL formulas in optimization-based approaches to falsification

Industrial Temporal Logic Specifications for Falsification of Cyber-Physical Systems

In this benchmark proposal, we present a set of large specifications stated in Signal Temporal Logic (STL) intended for use in falsification of Cyber-Physical Systems. The main purpose of the benchmark is for tools that monitor STL specifications to be able to test their performance on complex specifications that have structure similar to industrial specifications. The benchmark itself is a Git repository which will therefore be updated over time, and new specifications can be added. At the time of submission, the repository contains a total of seven Simulink requirement models, resulting in 17 generated STL specifications

Enhancing Temporal Logic Falsification with Specification Transformation and Valued Booleans

Cyber-Physical Systems (CPSs) are systems with both physical and software components, for example cars and industrial robots. Since these systems exhibit both discrete and continuous dynamics, they are complex and it is thus difficult to verify that they behave as expected. Falsification of temporal logic properties is an approach to find counterexamples to CPSs by means of simulation. In this paper, we propose two additions to enhance the capability of falsification and make it more viable in a large-scale industrial setting. The first addition is a framework for transforming specifications from a signal-based model into Signal Temporal Logic. The second addition is the use of Valued Booleans and an additive robust semantics in the falsification process. We evaluate the performance of the additive robust semantics on a set of benchmark models, and we can see that which semantics are preferable depend both on the model and on the specification